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Abstract 

In some contexts, well-formed natural language 
cannot be expected as input to information or 
communication systems. In these contexts, the 
use of grammar-independent input (sequences of 
uninflected semantic units like e.g. language- 
independent icons) can be an answer to the users' 
needs. However, this requires that an intelligent sys- 
tem should be able to interpret this input with rea- 
sonable accuracy and in reasonable time. Here we 
propose a method allowing a purely semantic-based 
analysis of sequences of semantic units. It uses 
an algorithm inspired by the idea of "chart pars- 
ing" known in Natural Language Processing, which 
stores intermediate parsing results in order to bring 
the calculation time down. 

Introduction 

As the mass of international communication and ex- 
change increases, icons as a mean to cross the lan- 
guage barriers have come through in some specific 
contexts of use, where language independent sym- 
bols are needed (e.g. on some machine command 
buttons). The renewed interest in iconic communi- 
cation has given rise to important works in the field 
of D esign (Archer and Krampen, 1996|; D reyfuss, 
1984; pta, 1993 ), on reference b ooks on the his tory 
and developm ent of the matter (Frutiger, 1991 ; Li- 
ungman, 1995; Sassoon and Gaur, 1997 ), as well 
as newer studies in the fields of Human-Computer 
Inter action and Digital Media (Yaz dani and Barker, 
2000) or Semiotics (IVaillant, 19991). 



We are here particularly interested in the field 
of Information Technology. Icons are now used 
in nearly all possible areas of human computer in- 
teraction, even office software or operating sys- 
tems. However, there are contexts where richer 
information has to be managed, for instance: Al- 
ternative & Augmentative Communication systems 
designed for the needs of speech or language im- 



paired people, to help them communicate (with icon 
languages like Minspeak, Bliss, Commun-I-Mage); 
Second Language Learning systems where learn- 
ers have a desire to communicate by themselves, 
but do not master the structures of the target lan- 
guage yet; Cross-Language Information Retrieval 
systems, with a visual symbolic input. 

In these contexts, the use of icons has many ad- 
vantages: it makes no assumption about the lan- 
guage competences of the users, allowing impaired 
users, or users from a different linguistic back- 
ground (which may not include a good command of 
one of the major languages involved in research on 
natural language processing), to access the systems; 
it may trigger a communication-motivated, implicit 
learning process, which helps the users to gradu- 
ally improve their level of literacy in the target lan- 
guage. However, icons suffer from a lack of expres- 
sive power to convey ideas, namely, the expression 
of abstract relations between concepts still requires 
the use of linguistic communication. 

An approach to tackle this limitation is to try to 
"analyse" sequences of icons like natural language 
sentences are parsed, for example. However, icons 
do not give grammatical information as clues to au- 
tomatic parsers. Hence, we have defined a method 
to interpret sequences of icons by implementing the 
use of "natural" semantic knowledge. This method 
allows to build knowledge networks from icons as 
is usually done from text. 

The analysis method that will be presented here is 
logically equivalent to the parsing of a dependency 
grammar with no locality constraints. Therefore, 
the complexity of a fully recursive parsing method 
grows more than exponentially with the length of 
the input. This makes the reaction time of the sys- 
tem too long to be acceptable in normal use. We 
have now defined a new parsing algorithm which 
stores intermediate results in "charts", in the way 
chart parsers ( Earley, 1970 ) do for natural language. 



1 Description of the problem 

Assigning a signification to a sequence of informa- 
tion items implies building conceptual relations be- 
tween them. Human linguistic competence consists 
in manipulating these dependency relations: when 
we say that the cat drinks the milk, for example, we 
perceive that there are well-defined conceptual con- 
nections between 'cat', 'drink', and 'milk' — that 
'cat' and 'milk' play given roles in a given process. 



Symbolic formalisms in AI (Sowa, 1984) reflect this 
approach. Linguistic theories have also been devel- 
oped spe^hicjillyto give account of these phenom - 
ena ( [Tesniere, 1959) ; |Kunze, 19751 ; |Mel'cuk, 19881 ), 
and to describe the transition between semantics and 
various levels of syntactic description: from deep 
syntactic structures which actually reflect the se- 
mantics contents, to the surface structure whereby 
messages are put into natural language. 

Human natural language reflects these conceptual 
relations in its messages through a series of linguis- 
tic clues. These clues, depending on the particular 
languages, can consist mainly in word ordering in 
sentence patterns ("syntactical" clues, e.g. in En- 
glish, Chinese, or Creole), in word inflection or suf- 
nxation ("morphological" clues, e.g. in Russian, 
Turkish), or in a given blend of both (e.g. in Ger- 
man). Parsers are systems designed to analyse nat- 
ural language input, on the base of such clues, and to 
yield a representation of its informational contents. 
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Syntactical analysis 
based on word order 



accusative : agent 
nominative : object 



Morphological analysis 
based on word inflexion 



In contexts where icons have to be used to convey 
complex meanings, the problem is that morpholog- 
ical clues are of course not available, when at the 
same time we cannot rely on a precise sentence pat- 
tern. 

We thus should have to use a parser based on 
computing the dependencies, such as some which 



have been written to cope with variable-word-order 
languages ( Covington, 199Q ). However, since no 
morphological clue is available either to tell that an 
icon is, e.g., accusative or dative, we have to rely on 
semantic knowledge to guide role assignment. In 
other words, an icon parser has to know that drink- 
ing is something generally done by living beings on 
liquid objects. 

2 The semantic analysis method 

The icon parser we propose performs semantic anal- 
ysis of input sequences of icons by the use of an 
algorithm based on best-unification: when an icon 
in the input sequence has a "predicative" structure 
(it may become the head of at least one dependency 
relation to another node, labeled "actor"), the other 
icons around it are checked for compatibility. Com- 
patibility is measured as a unification score between 
two sets of feature structures: the intrinsic semantic 
features of the candidate actor, and the "extrinsic" 
semantic features of the predicative icon attached 
to a particular semantic role (i.e. the properties "ex- 
pected" from, say, the agent of kiss , the direct object 
of drink , or the concept qualified by the adjective 
fierce ). 

The result yielded by the semantic parser is the 
graph that maximizes the sum of the compatibilities 
of all its dependency relations. It constitutes, with 
no particular contextual expectations, and given 
the state of world knowledge stored in the iconic 
database in the form of semantic features, the "best" 
interpretation of the users' input. 

The input is a sequence of icons s\, ... s n , 
each of which has a set of intrinsic features: 

IF(si) = Fi 
(where Fi is a set of simple Attribute- Value seman- 
tic features, used to represent intrinsic features of 
the concept — like {<human, +l>,<male, +1>} 
for Daddy). 

Some of the symbols also have selectional fea- 
tures, which, if grouped by case type, form a case 
structure: 

CS(si) = {{c 1 ,Fn},{c2,Fi 2 },... {c n ,F in )} 
(where each of the n cj is a case type such as 
agent, object, goal..., and each Fij a set of sim- 
ple Attribute- Value semantic features, used to deter- 
mine what features are expected from a given case- 
filler — e.g. <human , + 1 > is a feature that the agent 
of the verb write should possess). 

Every couple (cj,Fij) present in the case struc- 
ture means that is a set of Attribute- Value cou- 



pies which are attached to Sj as selectional features 
for the case Cj : 

SJ r (s i ,c j ) = Fij <S=^ (cj,Fij) £ CS(si) 

For example, we can write: 

SJ-(write,agent)={<human, +1>} 

The semantic compatibility is the value we seek 
to maximize to determine the best assignments. 

1 . At the feature level (compatibility between two 
features), it is defined so as to "match" extrinsic and 
intrinsic features. This actually includes a somehow 
complex definition, taking into account the mod- 
elling of conceptual inheritance between semantic 
features; but for the sake of simplicity in this pre- 
sentation, we may assume that the semantic com- 
patibility at the semantic feature level is defined as 
in Eq. |l|, which would be the case for a "flat" ontol- 

ogyQ- 

2. At the feature structure level, i.e. where the se- 
mantic contents of icons are defined, semantic com- 
patibility is calculated between two homogeneous 
sets of Attribute- Value couples: on one side the se- 
lectional features attached to a given case slot of the 
predicate icon — stripped here of the case type — , on 
the other side the intrinsic features of the candidate 
icon. 

The basic idea here is to define the compatibility 
as the sum of matchings in the two sets of attribute- 
value pairs, in ratio to the number of features being 
compared to. It should be noted that semantic com- 
patibility is not a symmetric norm: it has to measure 
how good the candidate actor fills the expectations 
of a given predicative concept in respect to one of its 
particular cases. Hence there is a. filtering set (S T) 
and a filtered set (TJ 7 ), and it is the cardinal of the 
filtering set which is used as denominator: 

C{ZT,ST) = C({/ n ,...,/ lm },{/ 21 ,...,/ 2n }) 

_ Ejg[l,n] Eie[Vnj C(fu, hj) 

n 

(where the fu and the are simple features of the 
form (au,vu) and (a,2j, V2j), respectively). 

A threshold of acceptability is used to shed out 
improbable associations without losing time. 

Even with no grammar rules, though, it is neces- 
sary to take into account the distance between two 

'The difference in computing time may be neglected in the 
following reasoning, since the actual formula taking into ac- 
count inheritance involves a maximum number of computing 
steps depending on the depth of the semantic features ontology, 
which does not vary during the processing. 



icons in the sequence, which make it more likely 
that the actor of a given predicate should be just be- 
fore or just after it, than four icons further, out of its 
context. Hence we also introduce a "fading" func- 
tion, to weight the virtual semantic compatibility of 
a candidate actor to a predicate, by its actual dis- 
tance to the predicate in the sequence: 

V(si,Cj,s k ) = D(s i ,s k ).C(TJ r (s k ),SJ r (s i ,Cj)) 

(3) 

where: 

V(si, Cj, s k ) is the value of the assignment of can- 
didate icon s k as filler of the role cj of predicate Si ; 

D is the fading function (decreasing from 1 to 
when the distance between the two icons goes from 
to oo); 

and C{TT{s k ),SJ : {si : Cj)) the (virtual) semantic 
compatibility of the intrinsic features of s k to the 
selectional features of Si for the case Cj, with no 
consideration of distance (as defined in Eq. ||). 

3. Eventually a global assignment of actors (cho- 
sen among those present in the context) to the case 
slots of the predicate, has to be determined. An as- 
signment is an application of the set of icons (other 
than the predicate being considered) into the set of 
cases of the predicate. 

The semantic compatibility of this global assign- 
ment is defined as the sum of the values (as defined 
in Eq. ||) of the individual case-filler allotments. 

4. For a sequence of icon containing more than 
one predicative symbol, the calculus of the assign- 
ments is done for every one of them. A global in- 
terpretation of the sequence is a set of assignments 
for every predicate in the sequence. 

3 Complexity of a recursive algorithm 

In former works, this principle was implemented by 
a recursive algorithm (purely declarative PROLOG). 
Then, for a sequence of N concepts, and supposing 
we have the (mean value of) V (valency) roles to 
fill for every predicate, let us evaluate the time we 
need to compute the possible interpretations of the 
sequence, when we are in the worst case, i.e. the N 
icons are all predicates. 

1. For every assignment, the number of seman- 
tic compatibility values corresponding to a single 
role/filler allotment, on an (actor, candidate) cou- 
ple (i.e. at the feature structure level, as defined in 
Eq. |) is: (N - 1) X V. 

2. For every icon, the number of possible assign- 
ments is: 



C((a 1 ,u 1 ),(a 2 ,U2» = ifai^a 2 

C((a, f i), (a, i; 2 }) = +1 if v\ and t/ 2 are equal integers 

— 1 if v 1 and v% are distinct integers 

v\ .V2 if one of the values is real 



N-l 



(N-iy. 



~ v (N-i-vy. (4) 

(we suppose that N — 1 > V, because we are 
only interested in what happens when N becomes 
big, and V typically lies around 3). 

3. For every assignment, the N — 1 allotment pos- 
sibilities for the first case are computed only once. 
Then, for every possibility of allotment of the first 
case, the N — 1 possibilities for the second case are 
recomputed — hence, there are (N— l) 2 calculations 
of role/filler allotment scores for the second case. 
Similarly, every possible allotment for the third case 
is recomputed for every possible choice set on the 
first two cases — so, there are (N — l) 3 computa- 
tions on the whole for the third case. This goes on 
until the V th case. 

In the end, for one single assignment, the num- 
ber of times a case/filler score has been computed is 
ELi (N - l) k . 

Then, to compute all the possible interpretations: 

1. Number of times the system computes every 
possible assignment of the first icon: 1. 

2. Number of times the system computes every 
possible assignment of the second icon: N ^ 1 Py 
(once for every assignment of the first icon, back- 
tracking every time — still supposing we are in the 
worst case, i.e. all the assignments pass over the 
acceptability threshold). 

3. Number of times the system computes every 
possible assignment of the third icon: N ~ l Py x 
N ~ l Py (once for every possible assignment of the 
second icon, each of them being recomputed once 
again for every possible assignment of the first 
icon). (...) 

4. Number of times the system computes 
every possible assignment of the N th icon: 

{ N-lp v) N-l_ 

5. Number of assignments computed on the 
whole: every assignment of the first icon (there 
are l Py of them) is computed just once, since 
it is at the beginning of the backtracking chain; 
every assignment of the second icon is computed 
N ~ x Py times for every assignment of the first icon, 
so ( Ar_1 Pv') 2 times, ... every assignment of the 
N th icon is computed ( N ~ l Py) N times. 



Total number of assignment calculations: 

V*iV /N-l p \k 

6. Every calculation of an assignment value in- 
volves, as we have seen, 

YX=i ( n - l) fe calcula- 
tions of a semantic compatibility at a feature struc- 
ture level. So, totally, for the calculation of all pos- 
sible interpretations of the sentence, the number of 
such calculations has been: 



V N 
k=l k=l 



7. Lastly, the final scoring of every interpretation 
involves summing the scores of the N assignments, 
which takes up N — 1 elementary (binary) sums. 
This sum is computed every time an interpretation 
is set, i.e. every time the system reaches a leaf of 
the choice tree, i.e. every time an assignment for the 
N th icon is reached, that is ( N ~ l Py) N times. So, 
there is an additional computing time which also is 
a function of N, namely, expressed in number of 
elementary sums: 

(N — 1) x (N-ipyf 

Hence, if we label a the ratio of the computing 
time used to compute the score of a role/filler allot- 
ment to the computing time of an elementary binary 
sum0, the number of elementary operations involved 
in computing the scores of the interpretations of the 
whole sequence is: 

(N-l).( N -'Py) N +a 2 (N - If. £ {^Pyf 
k=l k=l 

(5) 

4 The chart algorithm 

To avoid this major impediment, we define a new 
algorithm which stores the results of the low-level 
operations uselessly recomputed at every backtrack: 

'a is a constant in relation to N: the computation of the 
semantic compatibility at the feature structure level, defined in 
Eq. ^| roughly involves n x m computations of the semantic 
compatibility at the feature level, defined in Eq. [l] (n being the 
average number of selectional features for a given role on a 
given predicate, and m the average number of intrinsic features 
of the entries in the semantic lexicon), which itself involves a 
sequence of elementary operations (comparisons, floating point 
number multiplication). It does not depend on N, the number 
of icons in the sequence. 



a. The low-level role/filler compat- 
ibility values, in a chart called 
'compatibility_table'. The val- 
ues stored here correspond to the values 
defined at Eq. ||. 

b. The value of every assignment, in 
'assignments_table'. The values 
stored here correspond to assignments of 
multiple case slots of a predicate, as defined 
at point 3 of Section |2[ they are the sum of 
the values stored at level (a), multiplied by a 
fading function of the distance between the 
icons involved. 

c. The value of the interpretations of the sentence, 
in 'interpretations_table'. The val- 
ues stored here correspond to global interpre- 
tations of the sentence, as defined at point 4 of 
Section ^[ 

With this system, at level (b) (calculation of the 
values of assignments), the value of the role/filler 
couples are re-used from the compatibility table, 
and are not recomputed many times. In the same 
way, at level (c), the computation of the interpre- 
tations' values by adding the assignments' values 
does not recompute the assignments values at every 
step, but simply uses the values stored in the assign- 
ments table. 

Furthermore, the system has been improved for 
the cases where only partial modifications are done 
to the graph, e.g. when the users want to perform 
an incremental generation, by generating the graph 
again at every new icon added to the end of the se- 
quence; or when they want to delete one of the icons 
of the sequence only, optionally to replace it by an- 
other one. In these cases, a great part of the infor- 
mation remains unchanged. To take this property 
into account, the system stores the current sequence 
and the charts resulting from the parse in memory, 
allowing them to be only partially replaced after- 
wards. 

Finally, we have implemented three basic inter- 
face functions to be performed by the parser. The 
first one implements a full parse, the second par- 
tially re-parses a sequence where new icons have 
been added, the third partially re-parses a sequence 
where icons have been removed. The three func- 
tions can be described as follows. 

Parsing from scratch: 

1. Spot the icons in the new sequence which 



are potential predicates (which have a valency 
frame). 

2. Run through the sequence and identify every 
possible pair ((predicate,role), candidate). 

For each one of them, calculate the semantic 
compatibility 

C(2\F(candidate) , <SjF(predicate,role)). 

Store all the values found in 

compatibility_table: 



predicate 1 


role 1 


candidate 1 


value 


predicate 1 


role 1 


candidate 2 


value 


predicate k 


role V 


candidate N 


value 



(and eliminate values under the threshold as 
soon as they appear). 

3. Go through the sequence and identify the set of 
possible assignments for each predicate. 

For every assignment, compute its 
score using the values stored in 
compatibility_table, and multi- 
plying by the fading coefficients D(l), D(2), 

Store the values found in: 
assignments_table (Tab. [T]). 

4. Calculate the list of all the possible interpreta- 
tion (1 interpretation is 1 sequence of assign- 
ments). Store them along with their values in 

interpret at ion s_t able. 

Add a list of icons to the currently stored se- 
quence: 

1 . Add the icons of list of icons to the currently 
stored sequence. 

2. For every pair ( (predicate,role) .candidate). 

where either the predicate, or the candidate, is 
a new icon (is a member of list of icons), cal- 
culate the value of 

C(Z^ r (candidate) , 5/"(predicate,role)). 

and store the value in: 

compatibility_table. 

3. Calculate the new assignments made possible 
by the new icons from list of icons: 

• the assignments of new predicates; 



( 


predicate 1 


{ { role 1 , candidate /n 


(!))>• 


. ( role V, candidate f u (V) ) } ) 


value 


( 


predicate 1 


{ ( role 1 , candidate /12 


(!))>• 


. ( role V , candidate f l2 (V) ) } ) 


value 


( 


predicate k 


{ ( role 1 , candidate fk n 


(1)>.. 


. . ( role V , candidate f kn (V) ) } ) 


value 



Table 1 : Assignments Table 



• for every predicate already present in the 
sequence before, the assignments where 
at least one of the roles is allotted to one 
of the icons of list of icons. 

For each of them, calculate its value, and store 
it in assignments_table. 

4. Recompute the table of interpretations totally 
(no get-around). 

Remove a list of icons from the currently stored 
sequence: 

1 . Remove the icons of list of icons from the se- 
quence stored in memory. 

2. Remove the entries of 

compatibility_table or 

assignments_table involving at least 
one of the icons of list of icons. 

3. Recompute the table of interpretations. 

5 Complexity of the chart algorithm 

First, let us evaluate the complexity of the algorithm 
presented in Section || assuming that only the first 
interface function is used (parsing from scratch ev- 
ery time a new icon is added to the sequence). 

In the worst case: the JV icons are all predicates; 
no possible role/filler allotment in the whole se- 
quence is below the threshold of acceptability. 

• For every predicate, every combination be- 
tween one single role and one single other 
icon in the sequence is evaluated: there 
are (JV — 1) X V such possible couples 

(actor, candidate). 

• Since there are (worst case) JV predicates, there 
are JV x (JV — 1) X V such combinations to 
compute for the whole sequence, in order to 
fill the compatibility table. 

• After the compatibility table has been filled, its 
values are used to compute the score of every 
possible assignment (of surrounding icons) for 
every predicate (to its case roles). Computing 



the score of an assignment involves summing 
V values of the compatibility table, multiplied 
by a value of the fading function D, typically 
for a small integer. Thus, for every line in 
the assignments table (Table [[]), the computing 
time is constant in respect to JV. 

• For every predicate, there are 

N-lp (^-1)' 

V (N-l-V)l 

possible assignments (see Section ||). Since 
there are JV predicates, there is a total num- 
ber (in the worst case) of JV x N ~ 1 Py differ- 
ent possible assignments, i.e. different lines 
to fill in the assignments table. So, the time 
to fill the assignment table in relation to JV is 
iV! / (JV — 1 — V)\ multiplied by a constant fac- 
tor. 

• After the assignments table has been filled, its 
values are used to compute the score of the 
possible interpretations of the sentence. The 
computation of the score of every single inter- 
pretation is simply a sum of scores of assign- 
ments: since there possibly are JV predicates, 
there might be up to JV figures to sum to com- 
pute the score of an interpretation. 

• An interpretation is an element of the cartesian 
product of the sets of all possible assignments 
for every predicate. Since every one of these 
sets has N ~ l Py elements, there is a total num- 
ber of 

(N-i p \N (N-iy N 
1 v) (N-i-vy. N 

interpretations to compute. As each compu- 
tation might involve JV — 1 elementary sums 
(there are JV figures to sum up), we may con- 
clude that the time to fill the interpretations ta- 
ble is in a relation to JV which may be written 
so: (JV - 1) x ( N ~ l P v ) N . 

• In the end, the calculation time is not the prod- 
uct, but the sum, of the times used to fill each 



of the tables. So, if we label a and b two con- 
stants, representing, respectively, the ratio of 
the computing time used to get the score of 
an elementary role/filler allotment to the com- 
puting time of an elementary binary addition, 
and the ratio of the computing time used to get 
the score of an assignment from the scores of 
the role/filler allotments (adding up V of them, 
multiplied by values of the D function), to the 
computing time of an elementary binary addi- 
tion, the total computing time for calculating 
the scores of all possible interpretations of the 
sentence is: 

{N -\).{ N ~ l P v ) N + aVN{N -l) + bN{ N - 1 P v ) 

(6) 

6 Discussion 

We have presented a new algorithm for a completely 
semantic parse of a sequence of symbols in a graph- 
based formalism. The new algorithm has a tempo- 
ral complexity like in Eq. ||, to be compared to the 
complexity of a purely recursive algorithm, like in 
Eq.| 

In the worst case, the second function is still dom- 
inated by a function which grows hyperexponen- 
tially in relation to N: the number of possible in- 
terpretations multiplied by the time used to sum up 
the score of an interpretation^ In practice, the val- 
ues of the parameters a and b are fairly large, so 
this member is still small during the first steps, but 
it grows very quickly. 

As for the other member of the function, it is hy- 
perexponential in the case of Eq. 0, whereas it is of 
order bN^^Py), i.e. it is 0(N^ +1 ), in the case 
of Eq.g 

Practically, to make the semantic parsing algo- 
rithm acceptable, the problem of the hyperexponen- 
tial growth of the number of interpretations has to 
be eliminated at some point. In the system we have 
implemented, a threshold mechanism allows to re- 
ject, for every predicate, the unlikely assignments. 
This practically leaves up only a small maximum 
number of assignments in the assignments table, for 
every predicate — typically 3. This means that the 
number of interpretations is no longer of the or- 
der of N ~ 1 P V ) N , but "only" of 3^: it becomes 
"simply" exponential. This implementation mecha- 
nism makes the practical computing time acceptable 
when running on an average computer for input se- 
quences of no more than approximately 15 symbols. 

3 Namely, (N - 1).( N - 1 P V ) N . 



In order to give a comprehensive solution to the 
problem, future developments will try to develop 
heuristics to find out the best solutions without hav- 
ing to compute the whole list of all possible in- 
terpretations and sort it by decreasing value of se- 
mantic compatibility. For example, by trying to ex- 
plore the search space (of all possible interpreta- 
tions) from maximum values of the assignments, it 
may be possible to generate only the 10 or 20 best 
interpretations without having to score all of them 
to start with. 
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